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DIGITAL VIDEO SYSTEM FOR ASSEMBLING VIDEO SEQUENCES 

FIELD OF THE INVENTION 

The present invention relates to the capture and display of motion 
images and more particularly relates to an apparatus and method for producing 
5 more compelling video programs by a consumer. 

BACKGROUND OF THE INVENTION 
In the consumer market, steadily improving image quality, 
increased affordability, and ease of use have contributed to the rapid acceptance 
and use of video camcorders. Digital video capture and display devices have 

10 further benefited from advances in storage technologies using magnetic, optical, 
and electronic storage media such as digital videotape, memory cards, optical 
disks, and high-capacity random access memory (RAM) components. However, 
while camcorders have grown in popularity, many consumers express some level 
of disappointment with the image output achieved. For many reasons, the output 

1 5 from consumer camcorders can be disappointing or uninteresting to the viewer 
when compared with output captured and edited by imaging professionals. The 
problem is not typically caused by failure to use the equipment properly, but by a 
basic lack of a script, or a story, which is compelling to the viewer. Post- 
production modification and editing of video sequences is possible, but is 

20 inconvenient, and difficult to use for most consumers. 

Efforts have been made to address this problem by providers of 
digital image capture equipment and accessories. For example, commonly- 
assigned U.S. Patent No. 5,477,264 (Sarbadhikari et al.) discloses, for an 
electronic camera, a removable storage device preloaded with enhancement files 

25 for effecting camera operation, with additional graphical overlays and borders, 
and with built-in optimization, compression, and image enhancement algorithms. 
Commonly-assigned U.S. Patent No. 6,292,219 (Fredlund et al.) discloses an 
electronic motion picture camera system that provides special-effect video output. 
The device of U.S. Patent No. 6,292,219 provides a removable motion image- 

30 recording medium with programs that automatically, or as instructed by the 
camera operator, act upon a captured video stream to provide an edited 
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appearance, instructing the camera to affect imaging characteristics such as focus, 
highHghting effects, segues, fade-outs, and other effects. 

One technological advance of particular importance for imaging 
enhancement, and capitalized upon by the methods and apparatus of U.S. Patent 
5 No. 6,292,219 relates to the use of high-capacity random access storage device, 
such as an optical disc. Use of a random access storage device, as contrasted with 
accessing a continuous video stream stored on a cassette tape, provides the ability 
to insert pre-stored image sequences and effects at suitable points in the captured 
image sequence in order to display a pleasing motion picture presentation without 

1 0 abrupt scene changes. In place of an optical disc. Flash EPROM memory can be 
used instead. An example of an image capture apparatus incorporating a high- 
capacity Flash EPROM is the SD Multi AA^ Recorder, a ceimcorder from 
Panasonic, with U.S. headquarters in Secaucus, New Jersey. 

While the solutions disclosed in both the U.S. Patent Nos. 

15 5,477,264 and 6,292,219 provide, for example, the ability to improve the 

transitions between scenes, there remains significant room for improvement in 
making the captured scenes themselves interesting. For example, while these 
methods allow the use of pre-progranmied transitions, borders, and special effects, 
the consumer must make appropriate decisions in order to use these enhancements 

20 effectively. Further, solutions offered in these patents are not intended to provide 
the benefits afforded by professional editing techniques, such as scripting and 
composition used by cinematographers, who seldom shoot a motion picture 
sequence in the studio according to the timeline sequence of a motion picture. In 
motion picture studios, as is well known, scenes are shot according to a schedule 

25 that is best suited for production and are later assembled at an editing facility, 
according to a scripted storyboard. It can be appreciated that there would be 
advantages to providing some measure of storyboard capability to a consumer for 
event-based imaging with an edited appearance. The consumer would benefit 
from guidance in shooting appropriate scene content and would benefit from an 

30 ability to customize an image sequence, suited to the subject, audience, and event. 

Certainly, a substantial amount of motion video capture by 
consumers is event-based. Birthdays, holidays, family gatherings, parties. 
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graduations, wedding anniversaries, and the like are among the salient types of 
events that consumers wish to capture, retain, and display in a pleasing format. 
While the capabilities offered in U.S. Patent Nos. 5,477,246 and 6,292,219 
provide useful utilities and effects that can make electronic images from such 
5 events more pleasing, there remains a need for ways to help the consumer to 
capture images and image sequences that will display in a pleasing manner. 

Furthermore, there is an interest in using motion video capture 
devices to portray family and friends in a compelling or humorous way as part of 
a video program. For example, incorporating motion images of family or friends 
1 0 within a music video or parody show would provide an amusing program. 

Thus, there is a need to provide an electronic imaging system 
which includes instructions to enable a digital video system to obtain appropriate 
video sequences, and to automatically assemble the video sequences into a video 
program. 

1 5 SUMMARY OF THE INVENTION 

Briefly, according to one aspect of the present invention an image 
processing system for obtaining a plurality of image sequences and assembling a 
presentation from the plurality of image sequences comprises: 

(a) a camera for capturing the plurality of image sequences; 
20 (b) a random-access electronic memory for temporary storage 

of the plurality of image sequences; 

(c) an input reader for accepting pre-programmed instructions 
from a first storage medium; 

(d) a control panel comprising: 

25 (i) a display for viewing the image sequences and presenting 

pre-programmed instructions to the operator; and 
(ii) an operator interface for accepting operator responses to 
the pre-programmed instructions; 

(e) a control logic processor for executing the pre-programmed 
30 instructions and for arranging the presentation from the 

plurality of captured image sequences stored in the random- 
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access electronic memory according to the operator responses; 
and 

(f) an output writer for recording the presentation onto a second 
storage medium. 

5 According to another embodiment, the present invention provides a 

method for forming a presentation comprising a set of image sequences captured 
electronically, the method comprising: 

(a) obtaining programmed instructions for capturing members 
of the set of image sequences; 
10 (b) assembling an electronic storyboard, according to the 

programmed instructions, comprising a plan for the 
arrangement of the members of the set of image sequences; 

(c) prompting the camera operator to obtain each member of the 
set of image sequences by displaying operator instructions to 

1 5 the camera operator; 

(d) storing the set of image sequences in a memory; 

(e) assembling the presentation using the set of image 
sequences, according to the electronic storyboard; and 

(f) recording the presentation onto a storage medium. 
20 It is an object of the present invention to meet the needs for 

consumer capture of more pleasing images and image sequences electronically. 

It is an advantage of the present invention that it provides guidance 

to the camera operator by means of visual and/or audible instructions for capturing 

image content appropriate to a theme or event. 
25 It is a further advantage of the present invention that it frees the 

camera operator from editing or imaging sequence concerns, enabling the operator 

to have a more enjoyable experience in obtaining images electronically. 

It is a fiirther advantage of the present invention that it enables a 

novice camera operator to produce images and image sequences that have a more 
30 professional, edited appearance. 
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It is yet a further advantage of the present invention that it allows 
an operator to customize an event- or theme-based image sequence to be suited to 
a particular subject or audience. 

These and other objects, features, and advantages of the present 
5 invention will become apparent to those skilled in the art upon a reading of the 
following detailed description when taken in conjunction with the drawings 
wherein there is shown and described an illustrative embodiment of the invention. 
BRIEF DESCRIPTION OF THE DRAWINGS 
While the specification concludes with claims particularly pointing 
10 out and distinctly claiming the subject matter of the present invention, it is 
believed that the invention will be better understood from the following 
description when taken in conjunction with the accompanying drawings, wherein: 

Figure 1 is a schematic block diagram of an image capture and 
processing apparatus according to the present invention; 
1 5 Figure 2 is a perspective view of a camera used for image capture 

according to the method of the present invention; 

Figure 3 is a perspective view of the camera of Figure 2 in a 
supporting imaging dock assembly; 

Figure 4 is a plan view showing the appearance of the display of 
20 the camera for accepting preparatory information from the operator; 

Figures 5a-5d are plan views of the camera display showing a first 
example sequence of prompts for setting up a customized event-based image 
capture sequence; 

Figures 6a-6p are plan views showing a sequence of displays and 
25 prompts in one embodiment; 

Figure 7 is a perspective view of a camera providing a display and 
prompts, according to the present invention; 

Figure 8 is a flow chart showing a first example sequence of steps 
for obtaining images and assembling an electronic storyboard according to the 
30 present invention; 

Figure 9a is a block diagram showing an empty electronic 
storyboard, assembled according to the present invention; 
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Figure 9b is a block diagram showing a completed electronic 
storyboard, populated using images obtained according to the present invention. 

Figures 10a- lOg are plan views showing images obtained and 
processed according to the present invention; 
5 Figure 11 is a flow chart showing an example sequence of steps for 

capturing a music video based scripted program according to the present 
invention; 

Figures 12a-12e are plan views of the camera display showing a 
second example sequence of prompts for capturing a music video based scripted 
10 program; 

Figure 1 3 is a chart showing the relative timing of various 
components of audio and video components that are combined using the method 
of the present invention; 

Figures 14a- 14c are plan views of the camera display showing an 
15 example sequence for a humorous scripted presentation according to the present 
invention; and 

Figures 15a-15d are plan views of the camera display showing an 
example sequence for an altemate humorous scripted presentation according to the 
present invention. 

20 DETAILED DESCRIPTION OF THE INVENTION 

The present description is directed in particular to elements 
forming part of, or cooperating more directly with, apparatus in accordance with 
the invention. It is to be understood that elements not specifically shown or 
described may take various forms well known to those skilled in the art. 

25 Referring to Figure 1 , there is shown a schematic block diagram of 

an imaging apparatus 10 used for obtaining and processing still or motion picture 
images according to one embodiment of the present invention. Imaging apparatus 
10 has two major components, a camera 12 and an imaging dock assembly 30. 
Camera 12 consists of an image capture and recording apparatus 14 that includes 

30 imaging optics and electronic sensor circuitry, such as a conventional CCD, for 
example, for obtaining a two-dimensional electronic image. Image capture and 
recording apparatus 14 is supported by a RAM 16 and a display 18. A control 
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logic processor 20 provides operational logic for operation of camera 12 and 
cooperates with an interface circuit 22 for communication with imaging dock 
assembly 30 and with extemal input devices. 

Imaging dock assembly 30 has an input interface section 24 and an 
5 output interface section 26. Input interface section 24 accepts input software and 
data from any of a number of storage devices or input communication channels. 
Input data and software can be provided, for example, by magnetic or optical 
storage devices, such as floppy diskettes, memory cards, removable magnetic hard 
drives or optical media such as CDs and DVDs. Input can also be provided over 

10 wired or wireless ports, using any of a variety of networked or standalone 

communication mechanisms, such as Ethemet, IR, RF, or Bluetooth devices. 
Output interface section 26 may support audio input or playback circuitry 28, a 
printer 32, and a host of storage media. Image and/or audio output can be 
provided in the form of hard-copy output or in file form, using any number of 

1 5 wired or wireless communication and storage mechanisms. 

It must be emphasized that imaging apparatus 10 as shown in 
Figure 1 can have any of a number of alternative arrangements of components for 
camera 12 and its supporting imaging dock assembly 30. The basic components 
shown in Figure 1 are needed, in some form, for operation of the method of the 

20 present invention. Using the overall arrangement shown in Figure 1 , imaging 
apparatus 10 is capable of accepting programmed instructions and of capturing, 
processing, and playing back image and audio content according to those 
instructions, in a manner particularly well suited for event-based or theme-based 
imaging. 

25 Referring to Figure 2, there is shown, in perspective view, one 

embodiment of camera 12 with a removable storage device 34 in the form of a 
memory card. An operator interface panel 36 is provided for control of camera 12 
operation. Display 1 8 may be in the form of a touchscreen, enabling entry of 
additional operator instructions to camera 12. A stand 38 enables camera 12 to be 

30 seated on a surface for various image capture and playback functions. 

Referring to Figure 3, camera 12 is shown in position with imaging 
dock assembly 30. A CD/DVD port 40 is provided as an input/output mechanism 
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for receiving pre-programmed instructions as well as for receiving pre-stored 
audio and/or imaging content from an optical storage disk 42, such as a CD or 
DVD. In one embodiment, CD/DVD port 40 has read/write capability, allowing a 
captured image sequence, with optional audio content, to be written to a storage 
5 device for playback or archiving. Alternatively, other ports (not shown in Figure 
3) could be used for input and/or output to memory card or other storage devices. 

When operating according to the present invention, imaging 
apparatus 10 functions as a digital director, providing operator guidance and 
automated image processing utilities that enable even an unskilled operator to 

1 0 capture and construct a presentation of image sequences that can have the 
appearance of a professionally prepared production. As can be readily 
appreciated, imaging apparatus 10 would be especially well-suited for developing 
an image presentation that is event-based or theme-based. 
Obtaining Programmed Instructions and Images 

15 In order to function as a digital director, imaging apparatus 1 0 must 

first obtain a set of programmed instructions that are suited to the particular theme 
or event of interest. Additionally, other images and special effects can also be 
provided for digital director operation. In one embodiment, programmed 
instructions and images are available on a DVD or similar optically encoded 

20 medium. Various versions of digital director media could be made available to 
consumers, differentiated by theme or event. Typical activities for which digital 
director media could be particularly tailored include sports, hobbies, scout troops, 
camping and outdoor activities, art, music, and travel, for example. Themes for 
digital director media could include humor, romance, family gatherings, baby 

25 showers, babies and children, and the like. 

Other types of media could be used for providing or supporting 
digital director operation. Altematives include magnetic media of various types, 
such as diskettes. Flash memory cards or other types of memory cards could also 
be employed as digital director media. Still other altematives include 

30 commxmication of digital director instructions and images over wired or wireless 
connections, including the internet. 
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In effect, loading or downloading digital director instructions and 
images gives a "personality" to imaging apparatus 10 operation and editing, 
customizing how image and audio content is assembled to produce a presentation 
that is particularly suited to an event or theme. The example that follows shows 
5 one basic event of interest, a birthday party. Using the digital director, with the 
procedures described subsequently, an operator can film and produce a video 
presentation that is appropriately adapted for the age, sex, and interests of the 
subject. 

Operating Sequence - Preparation to Customize the Digital Director 

10 To operate the digital director, the consumer first loads digital 

director software into imaging apparatus 10. In the embodiment of Figure 3, 
electronic storyboard software and supporting images and effects load fi-om a 
DVD that acts as optical storage disk 42. As is shown in Figure 3, the operator 
feeds optical storage disk 42 into imaging dock assembly 30 as a preparatory step 

15 to image capture. Digital director software on optical storage disk 42 then 

prompts the consumer to make some preliminary selections that more precisely 
identify the event or theme for an image capture session. By way of example, and 
not of limitation, digital director prompts suited to a birthday theme may prompt 
the operator to identify age, sex, name, and other information about the person 

20 whose birthday is being celebrated. Prompts may appear on touchscreen display 
18 of camera 12, as is shown in the example of Figure 4. 

Figures 5a-5d show an example sequence with typical screen 
prompts that help to customize digital director operation, as is described 
subsequently. Touchscreen buttons 44 accept operator responses that help to 

25 tailor the sequence of operations that follow. An on-screen keyboard 46, as 

shown in Figure 5d, can be used to enable text entry. Altematively, if the camera 
is enabled with voice recognition capabilities, spoken words can be converted into 
text. Captured audio can also be used instead of text or to augment text. 

Referring to Figure 9a, there is shown, in conceptual form, an 

30 electronic storyboard 70, which is generated as a result of these initial preparation 
steps. Like its well-known equivalent in movie production, electronic storyboard 
70 specifies an ordered arrangement of image sequences. This can include 
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graphics 54 and title text 68 as well as planned segments 76, which are allotted for 
captured image sequences 64 when these are obtained. Transitions 66 are also 
provided between captured image sequences 64. These transitions may be fades, 
graphics, segues, or other scene transitions, using any of the techniques described 
5 in U.S. Patent No. 6,292,219, listed above or other familiar transitional 
mechanisms between captured image sequences 64, 

Electronic storyboard 70 as shown in Figure 9a is highly 
conceptual. In practice, any number of methods could be used for arranging 
captured image sequences 64 beforehand. However, Figure 9a shows the basic 

10 principles: a plan is generated for incorporating captured image sequences with 
pre-stored images and special effects loaded as part of the digital director data. In 
a simple embodiment, electronic storyboard 70 may be provided automatically, 
without the option to customize beforehand. However, in the more elaborate 
embodiment shown with reference to Figures 5a-5d, electronic storyboard 70 is 

1 5 customized, based on subject profile data entered by the operator. 

Image sequences, as the term is used in the present application, 
could be either motion or still images. As is well known, still images, if produced 
as part of a motion picture presentation, are simply repeated in multiple fi-ames. 
Additionally, still images may be presented in a pseudo motion manner such as 

20 that popularized by director Ken Bums in presentations such as "The Civil War," 
where the still image is slowly zoomed and panned to enhance its visual interest 
Operating Sequence - Operator Instructions from the Digital Director 

Figures 6a through 6o show an example sequence by which the 
digital director software guides the operator through individual image capture 

25 operations for obtaining image sequences that complete electronic storyboard 70. 

Referring to Figure 6a, display 1 8 provides instructions 48 and 
optional hints 50 to the operator. These may include requests for different types 
of captured images as well as tips for use of the camera and for lighting and other 
environmental conditions. As shown in Figure 6b, digital director may also 

30 provide messages 52 that give the operator various options for the image capture 
sequence. As shown in Figure 6c, added graphics 54 may also be suggested and 
provided for operator approval. A still image 56 may be inserted as one type of 
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image sequence if accepted by the operator, as is shown in Figure 6f. Graphic 
editing options 58 may also be provided, as is shown in the example of Figure 6g. 

The display examples shown in Figures 6h, 6i, and 6j show a 
"video karaoke" sequence that can be inserted. Using this feature, an operator 
5 follows instructions 48, which may provide branching options, as shown in the 
example of Figure 6h. As shown in Figure 6i, a sequence prompt 60 allows the 
operator to indicate when to begin capture of the video karaoke sequence. At this 
point, for example, the operator may wish to set camera 12 on stand 38, as is 
shown in Figure 7, and join in on the fun. Display 1 8 may then show the actual 

10 scene as a song or other sequence is performed or may simply display graphics 54, 
as shown in Figure 6j. During the karaoke sequence, an animated prompt 62 may 
display to guide participants through the text in an enjoyable manner. The output 
of the camera may be connected to a television so that the text is more legible. 
The captured video may or may not be displayed during this sequence. 

1 5 Figures 61, 6m, and 6n show various text graphics 54 added to 

scene content as it is captured and displayed on display 1 8. This example 
sequence is then completed with graphics 54 added to final scene content, as is 
shown in Figure 6p. To enhance the visual interest, it is preferred that the text 
graphics 54 move and/or change appearance as it is displayed, instead of being 

20 static. For example, in Figure 6p, the text "The End" can slowly fade in, the 

candles on the cake can flicker, and the balloons can slowly spin, and then "pop" 

at the end of the scene. 

Completing Electronic Storvboard 70 

Figure 9b shows the results of image capture and processing, in 

25 concept, according to instructions as shown in the example of Figures 6a-6p. 
With reference back to Figure 9a, electronic storyboard 70 is effectively 
transformed into a presentation 72 with captured image sequences 64 arranged in 
the order specified by electronic storyboard 70. Of course, operator responses 
during image capture may change electronic storyboard 70 dynamically; however, 

30 electronic storyboard 70 as initially created provides at least some structure for the 
prompt sequence and operation of imaging apparatus 10. Note also that image 
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sequences captured by the user were not necessarily captured in the order 
presented on electronic storyboard 70. 

Referring to Figure 8, a summary of steps carried out by control 
logic processing circuitry in imaging apparatus 10 for completing electronic 
5 storyboard 70 is shown. In an initiation step 100, the operator loads programmed 
instructions and, optionally, image content, and begins. In an optional prompt 
step 110, the operator is prompted for profile information on the subject to be 
filmed, as was shown in the example of Figure 5a. In an assemble storyboard step 
120, imaging apparatus 10 prepares electronic storyboard 70 or its equivalent. In 

10 an image capture step 130, the operator is prompted to film an image sequence, 
which may be a still or motion image, as is described above. A test step 140 
determines if all needed image sequences for completing electronic storyboard 70 
have been obtained and causes image capture step 130 to be repeated as necessary. 
Once this is true, an assembly step 150 processes the obtained image sequences 

15 and arranges them in the order assigned by electronic storyboard 70, forming 
presentation 72 thereby. 

During the assembly step 150, transitions between image 
sequences can use segues supplied as part of the digital director software, as 
described conmionly assigned U.S. Patent No. 6,292,219 (Fredlund et al.), the 

20 disclosure of which is incorporated herein by reference. Appropriate segues can 
be automatically chosen by the digital director to match the theme and to match 
the image sequences which begin and end at this particular transition. 

Alternatively, a partial completion of storyboard 70 may be used to 
for presentation 72, or a final editing step (not shown) can be provided for the user 

25 to approve and/or remove segments. A further embodiment allows the user to 
assign previously captured image sequences to storyboard 70, so that the user is 
not limited to use of image sequences captured under the guidance of digital 
director to create final presentation 72. Finally, in a recording step 160, imaging 
apparatus 10 writes presentation 72 onto a recording medium. In one 

30 embodiment, this is a DVD. This may be an unused, blank DVD or, optionally, 
the same DVD originally used to load programmed instructions in initiation step 
100. 
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Note that in an alternative embodiment, a control logic processor 
for executing the pre-programmed instructions and for arranging the playback of 
the presentation from the plurality of the captured image sequences stored in said 
random-access electronic memory according to said operator responses may not 
5 be in the camera. This function may be executed downstream of imaging 
apparatus 10 in a writer or computer (not shown). 
Special Effects 

Numerous special effects can be employed for further 
customization of presentation 72. These include, for example, slow motion, 

10 accelerated motion, auto-rotoscope, programmable time-lapse, time-reversal, 
image mirroring, and geometric distortion. Special effects can be applied at the 
time of image sequence capture or during final processing in preparing 
presentation 72. These effects are provided as part of the digital director software 
and can be automatically used for particular sequences. For example, an sequence 

15 of the "birthday boy" blowing out the candles on his birthday cake can be shown 
in the slow-motion "forward" (e.g. normal) direction, and then shown in the slow- 
motion "backward" direction, so that it appears that the candles are magically re- 
lighted. Similarly, by using "time-reversal," gifts can be re-wrapped and cakes 
can be re-assembled from cut pieces. 

20 Graphic templates may be provided for one or more image 

sequences, for both still and motion pictures. Templates may include decorative 
or thematic borders and could be populated with still or motion images using 
"drag and drop" or other suitable user interface techniques. A template could also 
allow display of multiple images at one time. Templates may also be used to 

25 create multiple still image formats for generating printed output, such as 

photographic quality prints, album pages, calendar pages, index prints, photo 
books, cards, invitations and the like, or for digital slide shows. Other graphic 
content, such as avatars and overlays, could also be used. 

Auxiliary sound clips could also be added, including sound clips 

30 provided with digital director media. Using auxiliary sound clips, for example, 
laugh tracks, music, sound effects, and other audio content could be mixed in 
with, or replace, audio content obtained during filming. 
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As can be seen from the above description, the digital director 
takes advantage of the capabilities afforded by digital image capture and random- 
access storage of image sequences and allows effective use of imaging tools 
disclosed in both U.S. Patent Nos. 6,292,219 and 5,477,264, while adding 
5 prompt/response sequences and automatic assembly of image sequences that both 
customize and simplify the image capture process. Digital director capability can 
be effectively used to guide a relatively unskilled operator through the sequence of 
steps necessary to produce a pleasing and effective theme- or event-based video 
presentation. The digital director frees the operator from timing and sequencing 

10 concerns when obtaining image sequences, allowing the operator to concentrate 
on one sequence at a time. By arranging the image content according to an 
electronic storyboard and adding transitional and special effects graphics element, 
the digital director helps to provide an amateur camera user with an output that 
has the advantages and appearance of being professionally edited. In some 

1 5 embodiments, the operator can immediately review the sequence that has just been 
captured along with any special effects, text graphics, etc. that will be used in the 
final presentation. If the final sequence is not visually pleasing, the operator can 
immediately recapture a new sequence while the subjects being photographed are 
still available. 

20 Second embodiment - Music video script 

In a second embodiment, a music video is used as the script to 
enable the user to produce a compelling program. The digital video camera is 
provided with digital director programmed instructions in the form of an audio 
soundtrack (a popular song, for example) a video track (such as music video 

25 scenes, for example) and a control track that provides instructions for video frame 
compositing and pseudo-camera movements. The audio soundtrack is played 
back as new video sequences are captured, so that the filmed subjects can act in 
time to the music. The captured live video is then combined with the video track 
to produce a new video sequence that includes both live images and the music 

30 video scenes provided by the digital director. 

In a preferred embodiment, the live images are captured with the 
subjects moving in front of a stationary background, allowing an altemate 
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backgroxmd to be added once the live images have been obtained. For this type of 
image processing, one or more initial images of background content only are first 
obtained and stored. This allows the background to be subsequently masked and 
replaced by a different moving background sequence provided by the video track, 
5 as described in commonly-assigned U.S. Patent No. 5,914,748 (Parulski et al.), the 
disclosure of which is incorporated herein by reference. 

The basic example series of image frames shown in Figures 10a- 
1 Og show how compositing of image frames 80 can be performed. A background 
image 82 is first obtained from the camera 12 position as shown in Figure 10a. A 

1 0 stored image 84, loaded for the digital director, provides the background desired 
for the edited music video sequence, as shown in Figure 10b. As is shown in 
Figure 10c, one or more subjects 86 are then captured, against the same 
background of Figure 10a. Following capture of subjects 86, image capture and 
recording apparatus 14 (Figure 1) then replaces background image 82 with stored 

1 5 image 84, with masking techniques applied to subjects 86, thereby forming a 
composited image 88, as is shown in Figure lOd. Alternatively, a user captured 
image sequence can be used to replace background image 82. However, the final 
composite image 88 may not be as convincing Figures lOe, lOf, and lOg show a 
zoom sequence executed on composited image 88 by image capture and recording 

20 apparatus 14. 

In a preferred embodiment, the control track provided by the digital 
director defines the way in which the foreground live image is composited into the 
background video track. The control track can include camera 12 control 
information that defines the camera 12 motion (such as zooming, panning, and 

25 rotation) of the background video track. This enables the same camera 12 motion 
to be digitally simulated in the live camera 12 image. For example, during a 
particular sequence, the background video can slowly pan to the left while 
zooming in. Using this technique, camera 12 itself can be stationary during the 
capture of the music video sequence, unchanged with respect to actual zoom 

30 position and other settings. The control track enables the digital video sequence 
captured by camera 12 to be digitally manipulated within image capture and 
processing apparatus 14 itself, thereby producing an apparent pan and zoom 
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sequence. Thus, without requiring sophisticated camera 12 controls or capture 
techniques, the digital director enables a visually interesting and realistic sequence 
to be produced. 

Referring to the flow chart of Figure 1 1 and to the series of display 
5 prompts in Figures 12a-12e, there is shown an example sequence of steps for 
capturing a music video based scripted program according to this embodiment of 
the present invention. A transfer step 200, initiated by a user selection for 
example, loads music audio to image capture and recording apparatus 14. A 
selection step 202, described in more detail subsequently, specifies a particular 

10 tune for playback during image capture. The camera 12 operator is provided with 
the procedures for setup in a setup instruction step 204. Camera 12 then captures 
background image 82 in a background capture step 206. With subjects 86 in 
place, a content capture step 208 is executed, obtaining images of subjects 86 
singing to the music. Following capture of this image content, image capture and 

15 recording apparatus 14 performs a number of processing steps. Each subject 86 is 
extracted from the actual background image during a background extraction step 
210. Panning, zoom, and other camera 12 effects are simulated in a simulation 
step 212. Each subject 86 is then combined with stored image 84 in a compositing 
step 214. Audio is added in a mixing step 216. The compositing step can include 

20 interspersing various background segments, provided as part of the digital director 
software, with the composite image segments. For the example shown in Figure 
10a- lOg, these background segments can show close-ups of the oars dipping in the 
water, the sun reflecting off the water, fish swimming in the water, etc. The final 
video production can then be recorded onto a suitable video storage medium, 

25 including a DVD provided with digital director software and content, for example. 

Display 18 of camera 12 provides prompts and allows operator 
selections to support the image capture portions of the sequence described with 
reference to Figure 1 1 . Figure 12a shows a selection prompt 90 for operator 
specification of song and overall theme, with a song selection button 92a for song 

30 selection and a theme selection button 92b for theme selection. Figure 12b shows 
a song selection menu 94a that displays when the operator presses song selection 
button 92a in Figure 12a. Figure 12c shows a theme selection menu 94b that 
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displays when the operator presses theme selection button 92b in Figure 12a. 
Figure 12d shows an initiation prompt 96 for beginning the music video sequence 
with setup instruction step 204 and following steps of Figure 1 1 . Figure 12e 
shows a lyrics prompt 98 that appears on display 1 8 during capture of the music 
5 video. Composited image 88 might be displayed dynamically, as is shown in 
Figure 12e. Altemately, display 18 may simply show subjects 86 as they are 
being captured, before image frame composition or display 18 may be limited to 
display of lyrics alone 

The timing chart of Figure 13 shows the temporal relationship of 

10 stored audio, shown as an audio file 170 in MP3 format, for example, with a 

stored background video sequence 172 and user video sequences 174. Based on a 
digital storyboard technique, the digital director generates a composited video 
sequence 1 76 that has segments showing background image segments 82 
interspersed with composited image segments 88. 

15 The result is a "personalized" music video, produced by the user 

and featuring a selected song, with one or more subjects 86 suitably featured as 
the "stars." The live scene is captured as the actors perform in accordance with 
the song, and the composited video images are stored in synchronization with the 
song. 

20 Third Embodiment — Humorous or Topical Scripted Sequence 

In a third embodiment, a random playback humorous scripted 
program or other topical sequence is produced. In this embodiment, the digital 
director provides instructions to the user to capture images of a number of 
participants. The participants are asked to answer a group of questions or to 

25 perform a group of actions in response to visual or audio prompts from camera 12 
or, more generally, from image capture and recording apparatus 14. The image of 
the participant is then captured while answering the questions and performing the 
actions as instructed. Each participant response segment is then available for 
playback by the digital director, controlled by a script. 

30 The display sequence of Figures 14a- 14c shows a simple example 

of a humorous sequence. As shown in Figure 14a, one or more suggestion 
prompts 1 80 may be posted to display 1 8, suggesting a scripted sequence. The 
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operator can accept the displayed sequence or select a different sequence. A 
series of prompts and instructions follows, as has been described with reference to 
the embodiments described above. In Figure 14b, subject 86 is posed a question 
or directed to execute an action. The response of subject 86 is recorded for 
5 playback, under control of the script that is selected. For example, in Figure 14c, 
video playback appears on display 18 of camera 12 itself. The context of the 
original question may be changed as shown, for example, for humorous effect. 
Example questions can elicit, for example, names of favorite foods, drinks, sports 
teams, activities, historical facts, and the like, for example. Prompted actions may 

10 include facial distortions, for example. Captured segments can be played back 
with interspersed narration. Special effects could be used, such as replay in the 
form of a news program, old movie, news reel, and the like, for example. 

Figures 15a-15d show another example using a humorous scripted 
sequence. After an affirmative response to suggestion prompts 180 in Figure 15a, 

15 images of various subjects 86 are captured and graphics added during image 

compositing, such as thought bubbles 182 shown in Figures 15b, 15c, and 15d, for 
example. These thought bubbles can be static text, or can be transparent graphic 
overlays that slowly become more visible. 

Nimierous other approaches could be used for execution of a script 

20 using the digital director. Editing, randomization, repetition, action reversal or 
"rewinding," and similar effects can be utilized to generate a suitable, compelling 
story line. More complex scripts could be employed, involving several subjects 
86 in different scenes of a story, such as a fairy tale, for example. The digital 
director utility could be used, for example, to generate a scripted tribute or 

25 greeting card, for example. Video sequences obtained using the digital director 
could be stored on various types of storage media, printed, and transferred to other 
display apparatus, for example. 

The digital director software can be provided as a computer 
program product. A computer program product can include one or more storage 

30 medium, for example; magnetic storage media such as magnetic disk (such as a 
floppy disk) or magnetic tape; optical storage media such as optical disk, optical 
tape, or machine readable bar code; solid-state electronic storage devices such as 
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random access memory (RAM), or read-only memory (ROM), or obtained via a 
communications link such as connection to a website via the Internet; or any other 
physical device or media employed to store a computer program having 
instructions for practicing a method according to the present invention. 

The invention has been described in detail with particular reference 
to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the scope of the invention as described 
above, and as noted in the appended claims, by a person of ordinary skill in the art 
without departing from the scope of the invention. For example, imaging 
apparatus 1 0 may have any number of different arrangements of the basic 
components necessary for implementing digital director functions. Read and 
write capability could be executed using a number of different media types and 
wired or wireless communication mechanisms. 

Thus, what is provided is an apparatus and method for enhancing 
image output from an electronic imaging device, using real-time instructions and 
scripted special effects. 
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PARTS LIST 

1 0 imaging apparatus 

12 camera 

14 image capture and recording apparatus 

16 RAM 

1 8 display 

20 control logic processor 

22 interface circuit 

24 input interface section 

26 output interface section 

28 audio input or playback circuitry 

30 imaging dock assembly 

32 printer 

34 removable storage device 

36 operator interface panel 

38 stand 

40 CD/DVD port 

42 optical storage disk 

44 touchscreen buttons 

46. keyboard 

48 instructions 

50 hints 

52 messages 

54 graphics 

56 still image 

58 graphic editing options 

60 sequence prompt 

62 animated prompt 

64 captured image sequence 

66 transition 

68 title text 

70 electronic storyboard 
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72 


presentation 


76 


planned segment 


80 


image frame 


82 


background image 


84 


stored image 


86 


siibject 


88 


composited image 


90 


selection prompt 


92a 


song selection button 


92b 


theme selection button 


94a 


song selection menu 


94b 


theme selection menu 


96 


initiation prompt 


98 


lyrics prompt 


100 


initiation step 


110 


prompt step 


120 


assemble storyboard step 


130 


image capture step 


140 


test step 


150 


assembly step 


160 


recording st^ 


170 


audio file 


172 


stored background video sequence 


174 


user video sequence 


176 


composited video sequence 


180 


suggestion prompt 


182 


thought bubble 


200 


transfer step 


202 


selection step 


204 


setup instruction step 


206 


background capture step 


208 


content capture step 
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210 background extraction step 

212 simulation step 

2 1 4 compositing step 

216 mixing step 



